A Bag of Semantic Words Model for Medical Content-based Retrieval

The bag of visual words model has been widely used in contentbased image retrieval. However, when it is applied to medical domain, it potentially has several limitations, e.g., some ordinary feature descriptors may not be able to capture the subtle characteristics of medical images; there is a semantic gap between the low-level features and the medical concepts; the emerging multi-modal data pose challenges on current retrieval framework and urge us to extend the possibilities to combine and analyze the multi-modal data. In an attempt to address these issues, we proposed a bag of semantic words model for medical content-based retrieval in this study. We built the high-level semantic features from the low-level visual features by a three-step pipeline. We first extracted a set of low-level features pertaining to the disease symptoms from the medical images. We then translated the low-level features to symptom severity degrees by symptom quantization. Finally, the high-level semantic words were built through learning the patterns of the symptoms. The proposed model was evaluated using 331 multi-modal neuroimaging datasets from the ADNI database. The preliminary results show that the proposed bag of semantic words model could extract the semantic information from medical images and outperformed the state-of-the-art medical content-based retrieval methods.